Facing The Machine Translation Babel in CLIR – Can MT Metrics Help in Choosing CLIR Resources?

This paper describes usage of MT metrics in choosing the best candidates for MT-based query translation resources in Cross-Language Information Retrieval. Our metrics is METEOR. Language pair of our evaluation is English → German, because METEORmetrics does not offer very many language pairs for comparison. English → German has also available many MT programs that can be used in evaluation. We evaluated translations of CLEF 2003 topics of twelve different MT programs with MT metrics and compare the metrics evaluation results to mean average precision results of CLIR runs. Our results show, that for long topics the correlations between achieved MAPs and MT metrics are high (0.88), and for short topics lower but still clear (0.59). Overall it seems that METEOR can easily distinguish the worst MT programs from the best ones, but smaller differences are not so clearly seen. Some of the intrinsic properties of METEOR metrics do not also suit for CLIR resource evaluation purposes, because some properties of the translation metrics, especially evaluation of word order, are not significant for CLIR resource evaluation.